Transmitting enhancement data for video

ABSTRACT

The invention provides a method and apparatus that addresses and resolves the issues currently affecting the ability to offer Enhanced TV, in particular, those issues concerning timing and synchronization, interaction with other modules in the STB, and distribution.

This application is a divisional of U.S. patent application Ser. No.10/407,090, filed Apr. 2, 2003, which claims the benefit of U.S.Provisional Application No. 60/370,024, filed Apr. 3, 2002. U.S. patentapplication Ser. No. 10/407,090 and U.S. Provisional Application No.60/370,024 are both incorporated in their entireties by referenceherein.

BACKGROUND OF THE INVENTION

1. Technical Field

The invention relates to information distribution. More particularly,the invention relates to a method and apparatus for transmitting data ina data stream.

2. Description of the Prior Art

The term “Enhanced TV” refers to the delivery of additional data to aclient beyond the current standards of audio, video, teletext, andsubtitle data. These data may be used to enhance the viewing experiencein many ways—they may be used to prompt the user to take certainactions, they may be used to place informational graphics on the screen,or they may be used to create truly interactive programming, where theviewer has a great deal of control over what appears on the screen. As amedium, Enhanced TV is still in its infancy, and many opportunities forapplying the technology have yet to be explored.

The technology of Enhanced TV is one of distributing a shared userexperience to many clients simultaneously. The user experience can bemore dynamic, flexible, and interactive than pure audio or videoexperiences. This technology can be used without audio or video, or withaudio or video alone.

There are several fundamental pieces of an Enhanced TV system. On thebroadcast or head end side, there is a source, which is often a serveror suite of servers that schedule and format the enhancement data. Thedata are then integrated into the audio and video data streams, and sentdown to the end users' homes through a broadcast transmission system.The essential action of the source is the creation of a set of data forlater distribution. Although the source is referred to in this documentas a server, the source may not be a server in the strict sense. Forinstance, the source could be a software component that writes files toa disk for later broadcast, or annotates an MPEG file, or creates one ormore proprietary files to be read by a transmission agent, such as aspooler.

On the end user side, there is a module in the television or set top boxclient in the viewer's home that interprets the enhanced TV data. Thismodule may carry out commands in the data itself, or it may launch adedicated application, referred to as an enhancement, which may have itsown methods of listening to and interpreting the enhanced TV data.

In developing Enhanced TV systems and Enhanced TV content/enhancements,several issues arise:

1) Synchronization and Timing. Many enhancements rely on very closesynchronization between the data stream as delivered to the enhancementand the audio/video shown to the user. For instance, a quiz showenhancement should not present questions until the on-screen announcerhad announced them. Even if the audio/video data and the enhanced TVdata are delivered to the set top box at the same time, they may takevery different data paths internally. Different delays in these datapaths create synchronization problems that are visible to the viewer.Delays may also come from other sources as well as the client, e.g.servers, networks, and transmission media all may introduce their owndelays.

2) Interactions with other modules in the client. In many of the moremodern and complex client systems, there are many modules besides theEnhanced TV module running concurrently. These modules may include anElectronic Program Guide (EPG), Video On Demand services, and billingsystems, among others. Defining how the Enhanced TV module and itssubservient enhancements interact with these other modules isnon-trivial because some, but not all, of the capabilities of theseother modules must be exposed to the Enhanced TV system. The picture isfurther complicated by the fact that different network operators havedifferent needs, and wish to expose different pieces of the full systemto the Enhanced TV subsystem. Network operators may additionally havedifferent business relationships with different enhanced contentproviders, and may wish to expose or hide parts of the system based onthose business relationships.

3) Distribution. Because Enhanced TV data is created to enhancetelevision broadcasts, Enhanced TV data originate from many differentsources, e.g. broadcasters and television content producers. Unlikeconventional television programming, multiple enhancements may be activeand consuming resources at the same time. It is important to be able tomake sure enhancements from distributed and mutually unaware sources beable to run in a single integrated environment.

4) Loading. Because Enhanced TV data are often not resident on theclient, and must be pulled off of a data stream, there is often a delaybefore the enhancement can be presented to the user. Furthermore,limitations of the client, server, and distribution mechanism may makeit impossible to run the enhancement correctly. Such limitations mayinclude memory or resource limitations on the client, and data deliveryerrors within the server and distribution mechanism. For the userexperience, it is important that some amount of functionality of theenhancement be guaranteed before any notification of the enhancement ispresented to the user.

It would be advantageous to provide a method and apparatus thataddresses and resolves the issues currently affecting the ability tooffer Enhanced TV, in particular, those issues concerning timing andsynchronization, interaction with other modules in the STB,distribution, and loading.

SUMMARY OF THE INVENTION

The invention provides a method and apparatus that addresses andresolves the issues currently affecting the ability to offer EnhancedTV, in particular, those issues concerning timing and synchronization,interaction with other modules in the STB, and distribution.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block schematic diagram of a system architecture accordingto the invention;

FIG. 2 is a block schematic diagram of a system architecture accordingto the invention when there is no broadcast data server;

FIG. 3 is a block schematic diagram of a system architecture accordingto the invention when the invention is applied to a Video On Demand(VOD) system;

FIG. 4 is a block schematic diagram of a system architecture accordingto the invention when the invention is applied to a Digital VideoRecorder (DVR) system;

FIG. 5 is a block diagram illustrating multiplexing of time pulses anddata associated with different substreams in one data stream;

FIG. 6 is a block diagram that illustrates the problem of accessing aprivileged function from an unprivileged document;

FIG. 7 is a block diagram that illustrates a technique for accessing aprivileged function from an unprivileged document according to theinvention; and

FIG. 8 is a block diagram that illustrates a technique for signalingbetween privileged and unprivileged contexts without granting securityaccess according to the invention.

DETAILED DESCRIPTION OF THE INVENTION

The invention provides a method and apparatus that addresses andresolves the issues currently affecting the ability to offer EnhancedTV, in particular, those issues concerning timing and synchronization,interaction with other modules in the client, and distribution. Theinvention is described in connection with an Enhanced TV system, butthose skilled in the art will appreciate that the invention may beapplied to other similar systems, such as recent cell phones andpersonal data assistants (PDAs), as well as general personal computing(PC) environments.

FIG. 1 is a block schematic diagram of a system architecture accordingto the invention.

At a high level, the presently preferred embodiment of the inventioncomprises a computer (server) 12 that schedules and transmits data—, asystem for distributing (transmission mechanism 16) that data to othercomputers (clients) 11, and one or more computers (clients) whichreceive the transmitted data and act on them in a defined fashion. Theseparts work together to for a single distributed computing system.

Also shown on FIG. 1 is an enhanced TV management system, i.e.schedulers 10, a content data server 15, and an auxiliarydata/video/audio server 19. These features of an enhanced TV system arewell known to those skilled in the art.

The computer that schedules and transmits is known as a broadcast dataserver 12. The computer that receives and interprets the data streams isknown as a broadcast client 11. The infrastructure for distributing thedata is known as the transmission mechanism 16. The broadcast dataserver 12 may be a single computer or it may consist of many computerscollaborating to create one or many broadcast data streams.

Note that the broadcast data server 12 may not be a server at all. It isany device, component, software or the like that provides thetransmission mechanism 16 with one or more data streams. The function ofthe broadcast data server 12 is the creation of data for distribution.The data may be sent through the transmission mechanism 16 immediatelyin a true server mode, or it may be stored in some way for broadcast orother transmission later. Such storage mechanisms may include thecreation of proprietary files that can be read by a spooler, orannotations to an MPEG file, among others. One embodiment of theinvention is shown in FIG. 2 (discussed below), where the data arestored in a unit that is marked Storage Media. For simplicity, thiscomponent of the invention is referred to as a “server” herein.

The broadcast data server 12 optionally communicates with a content dataserver 15, which is responsible for fetching or creating the contentassociated with the enhancement. A scheduler 10 is responsible forspecifying to the broadcast data server 12 which data are to be placedon which output data streams at which times.

The broadcast data server 12 is responsible for gathering the data,often with the assistance of the content data server 15, scheduling thedata so that they appear at the correct times on the output datastreams, and formatting the data in a way that is compatible with thetransmission mechanism 16. In the case of MPEG transmissions over a DSLconnection, the data can be formatted as MPEG transport streams, but itcan also be formatted as Internet Protocol IP packets in an IP basedphone system, or as data to be inserted in the Vertical BlankingInterval (VBI) in an analog TV system, or in some other way appropriateto the transmission mechanism in which the invention is embodied.

There may be multiple schedulers 10 feeding a single broadcast dataserver, as shown in FIG. 2. This allows multiple mutually independentenhancement providers to use the system simultaneously. For instance ina digital satellite TV system, many broadcasters can use separateschedulers but the same broadcast data server to provide enhancements ontheir individual TV channels. Similarly, a scheduler associated with anadvertisement insertion system can drive enhancements associated withadvertisements, while a separate scheduler can drive enhancementsassociated with the standard programming.

The scheduler 10 may also communicate with one or more optionalauxiliary audio, data, or video server 19. The auxiliary server 19 isresponsible for sending data that are not directly enhancement-relatedto the client, either through the same transmission mechanism used bythe enhancement, or through some other means. In the case of a digitalTV system or VOD service, auxiliary server 19 can be a standard videoserver. Communication with an auxiliary server, while not required, isuseful when synchronizing enhancement data with auxiliary data.

Note that although auxiliary server 19 is likely to be present in mostembodiments of this invention, its existence is not required. Many ofthe high value applications of the invention involve adding to existingtelevision infrastructure, and distributing an experience that enhancesthe traditional capabilities of television or radio. However, the systemmay present its own self contained experience.

Although the broadcast data server 12, the content data server 15, andthe scheduler 10 are shown as independent entities in the figures, thereis no requirement that they be so. They could be hosted on the samephysical piece of equipment, or even merged into a single integral pieceof software, or their functionality could be provided in a variety ofother ways and combinations.

Similarly, it is possible for multiple broadcast data servers to serve asingle set of clients. For instance, in a digital cable system, eachbroadcaster can have its own scheduler and broadcast data serversfeeding into its own MPEG data streams, which would be multiplexedtogether into the final transport stream at the cable operator's headend.

The presently preferred broadcast server 12 sends three distinct streamsfor each enhancement, i.e. an Announcement stream 100, a Trigger stream104, and a Content stream 102. These streams are shown on FIG. 1 as eachhaving a path from a scheduler 10, through the server 12, a distribution(transmission) mechanism 16, one or more clients 11 and the Enhanced TVservice 13. The Announcement stream 100 serves as a mechanism fornotifying the client of which enhancements are available. The Triggerstream 104 contains time-sensitive information for the enhancementitself, and also carries the time synchronization protocol data. TheContent stream 102 contains the resources that make up the enhancementsthemselves, or references that enable the retrieval of the enhancements.Such resources and references may include but are not limited to HTMLfiles, JavaScript, JAR files, GIFs, binary data, URLs and referenceidentifiers. These streams are transmitted to the client(s). Although itis preferred that most enhancements carry all three streams, it is notstrictly necessary.

Note that a trigger in this context does not refer only to EIA 746Adata. A trigger in as used herein can be any discrete chunk ofinformation defined by the enhancement, may be formatted in anyarbitrary way, and may consist of strings, numbers, binary data, andothers. These data arrive on the client as a unit, and can be marked forpresentation to the enhancement at a predefined time.

As an example, in an interactive TV system running over DSL lines, theserver could produce streams that are combined into the MPEG data feedthat is sent to the digital set top boxes or digitally enabled TVs. TheAnnouncement stream could contain tables of name value pairs, where thenames specify known attributes of the enhancement, such as the URL orother unique reference identifier for the enhancement to run, the startand stop time for the enhancement, and information on how to prompt theviewer to run the enhancement. The Trigger stream could containinformation relating to state changes in the enhancement, for instancename value pairs of “Question” and “Answer” relating to questionscurrently being asked on a quiz show. The Content stream could containthe data associated with the enhancement, including the root content (inthis example, possibly “quizshow.html”) and associated resources (forinstance, “questions.js”, “correct.gif”, etc).

In the above example, the streams could be formatted as MPEG privatesections, and are associated with particular video services throughadvertisement in the Program Map Table (PMT), or similar mechanism. Sucha mechanism allows the client to find the enhancements associated withstandard broadcast data.

Data for different enhancements may share streams, that is, there is norequirement that the streams for different enhancements be distinct. Inthe previous example, for instance, all enhancements for a particularMPEG/DVB service can share the same Announcement, Content, and Triggerstreams. There is also no requirement that enhancements must share thesame streams.

Data are transmitted through the transmission mechanism to one or moreclients, processed, and the enhancement displayed to the users Thetransmission mechanism is inherently digital, although the digitalsignal typically is encoded in analog signals. The transmissionmechanism can be of variable delay, and is commonly comprised of one ormany computers, devices, and data paths. An example of a low delaymedium is a digital television transmission system, which carriesdigital data along with the digital television system. An example of ahigh delay system is a system that encodes data on a digital video tape,and retransmission through a digital television transmission systemoccurs at a later time. Another example is a stock ticker distributionsystem based on Internet Protocol (IP) networking destined for portable,networked devices such as cell phones. The broadcast media itself can bean RF cable network, as in the preceding example, but can also be phonelines, Ethernet, satellite signals, fiber optics, or any other broadcasttransmission medium. It should be understood that the principles of thepresent invention apply not only to systems using broadcast transmissionmediums but also to systems employing pointcast, multicast and othertransmission mediums.

On the client, a software element, the enhancement service 13 in FIG. 1,listens to the Announcement stream, and processes the data that itreceives on this stream. Data on the Announcement stream containcommands to start and stop enhancements, and references to theenhancement being launched. Depending on the constraints in the clientitself, for instance whether the user wishes to see or otherwise accessthe enhancements, and constraints imposed by the network operator, forinstance whether broadcast owners of the currently tuned TV service havepaid to have their enhancements visible, the enhancement service canlaunch an enhancement.

An enhancement, in the terms of this invention, is a set of content thatcan be run (e.g., interpreted, executed or otherwise run) and displayedin the client environment. The “runable” (e.g., interpretable orexecutable) elements of an enhancement can be in program format, such asobject code, Java byte code or interpretable JavaScript; or they can bestructured as data for a presentation engine, such as html. The contentassociated with an enhancement can also contain binary data items fordisplay to the user, for example, GIFs, JPEGs, and WAVs. Some or all ofan enhancement's content may already be resident on the client when thecommand to start the enhancement comes in on the Announcement stream;some or all of an enhancement's content may be pulled in from theContent stream. For example, all of the data necessary to display anenhancement associated with a special one-time only broadcast, such asthe Academy Awards, are likely to be pulled in off the Content stream.However, core modules associated with a standard enhanced advertisingsystem that are used by many advertisements are likely to be locallystored in the updateable, re-writable compact flash memory in the settop box. Alternatively, core modules could be stored locally in otherset top box memory.

Launching an enhancement consists of loading the initial content neededby the enhancement onto the client if it is not already resident there,optionally displaying a prompt to the user and, if the user chooses toaccept the prompt, or if the user is not given a prompt, running theinitial interpretable (runable) elements or sending them to thepresentation engine, as appropriate.

An enhancement may be of long duration, such as the display of a flow ofstock information that lasts years, or short in duration, such as addinginteraction to a 30 second television commercial.

To listen to the Announcement stream, the client must have some way ofidentifying this stream. In the MPEG/DVB transport example, theAnnouncement stream is carried at a well-known location, for example aspecific PID, or advertised as a component with a well-known stream IDon the PMT. Other options in other systems include, for example, awell-known broadcast IP address, a well known frequency, or anindirection mechanism similar to the PMT in which a known physicalresource, such as PID, frequency, or IP address, contains a table whichmaps available streams or other resources to their physical locations.

The enhancement itself may “listen to” or otherwise monitor a Triggerstream. The problem of identifying the Trigger stream is similar to theproblem of identifying the Announcement stream, and can be solved bysimilar means. In addition, the physical resource containing the Triggerstream may be specified in the Announcement stream. In this case theenhancement may refer back to the Announcement that launched it, andfind the physical location there.

Both the enhancement and the program on the client responsible forlistening to Announcements and launching enhancements can use theContent stream. Because there may be multiple resources on a singleContent stream, a mechanism is needed to map individual resourcereferences to physical locations within the Content stream. Such amechanism is found in a Table of Contents (TOC) for the Content stream.The TOC for the Content stream is normally contained in theAnnouncements stream, to which the client is automatically listening.After receiving the TOO, the client uses the information in it toestablish a mapping between the names of the individual items on theContent stream and the subsections of the stream that contain thoseitems.

For example, in the case where the streams are contained within MPEGtransport streams, the TOC can establish the relationship between thereference (URL, URI, or other) that identifies the particular contentitem (“http://www.quizshow.com/quizshow.html”) and the table ID and/orextended table ID of that item within the Content stream (table ID 0x80,extended table ID Ox4ef8). If there is more than one Content streamavailable, there is additional information pertaining to which Contentstream the content item is on, for instance a known PID (PID Ox2e, tableID 0x80, extended table ID Ox4ef8), or other identifier for the Contentstream, i.e. 0x93-01-0x80-0x4ef8, where 0x93-01 identifies the streamwithin the service uniquely by stream type and component tag in the PMTdescriptors, or other mechanism specific to the particular transportused.

As additional examples, alternate specific embodiments of the inventionare given in FIGS. 2, 3, and 4.

In FIG. 2, the broadcast data server 12 if FIG. 1 is replaced by a datasource and storage medium 12 a, and data on the storage medium is thensent directly to the transmission mechanism 16.

FIG. 3 shows an embodiment of the invention in a VOD environment. Inthis case, the transmission mechanism consists of a VOD recorder 131,VOD storage device 133, and a VOD player 132. The broadcast data server12 and auxiliary video/audio server 19 feed into the VOD recorder 131,where the combined data are formatted and sent to into storage in theVOD storage device 133 for playback at a later time. The VOD player 132is responsible for sending this data to the client in response to aclient request. Note that for simplicity the Announcement, Trigger,Content, and Auxiliary data streams are represented by a single line inFIG. 3, however they are still separate streams and no significantdeparture from their use in FIG. 1 is implied or indicated.

FIG. 4 shows an embodiment of the invention in a Digital Video Recorder(DVR) environment. In this environment, the streams from thetransmission medium are stored for an indeterminate period of time inlocal storage 141 on the client, instead of being sent immediately tothe enhancement and enhancement service. Playback from local storagebegins on the user's command. Again, for simplicity, Announcement,Trigger, Content, and Auxiliary data streams are represented by a singleline, and again, no significant departure in these streams from thediagram in FIG. 1 is implied or indicated.

Distribution and Synchronization

Some types of information, for instance state information carried on aTrigger Stream, must be delivered to the enhancement on the client at aguaranteed time. Such a mechanism, for instance, allows tightsynchronization of events in a television broadcast with events withinthe enhancement.

In the presently preferred embodiment of the invention, a mechanism isprovided whereby timing information is transmitted from the server tothe client, and information on the client can be guaranteed to bedelivered to the enhancement at a particular time, according to thistiming information. This timing information is specific to a particularenhancement, allowing multiple enhancements to run simultaneously andindependently.

This mechanism comprises a means to set the state on the server, aprotocol for transmitting that state to the client, and a method for theclient to ensure that events are delivered at the appropriate times.

On the server, the main data streams are split into data substreams, onefor each enhancement, which can be addressed independently in commandsto the server. These data substreams can be created and destroyedprogrammatically, that is, in commands to the server, and may sharephysical resources within their main data stream. For instance, if theserver has an XML command interface, a command to create a datasubstream can have the following syntax:

<CAROUSEL NAME=“kqed”> <CREATEVIRTUAL NAME=“trigger5127> <DESTROYVIRTUALNAME=“trigger511”/> <CAROUSEL>

where the <CAROUSEL> tag identifies the main data stream, and the<CREATEVIRTUAL> and <DESTROYVIRTUAL> tags identify the data substreamsto be created and destroyed within that main data stream.

Timing state is set for a data substream by creating a virtual time basefor the substream. This time base is initialized by specifying anabsolute time and corresponding time base time, for instance,

<CAROUSEL NAME=“kqed” VIRTUAL=“trigger512”> <SYNCHRELATIVE=“0”UTC=“2001-12-20T02:30:00” CONTINUOUS=“false”> </CAROUSEL>

which sets/initializes the time base of substream “trigger512” to 0 atUTC time Dec. 12, 2001, 2:30:00. There is no requirement that time onthe virtual time base move forward in real time, that is, one second persecond, although it is anticipated that in many cases it will.

Note that absolute time may be specified by any number of differentmechanisms: UTC time, GMT time, Pacific Daylight Time, number of secondssince a defined epoch, among others. The only requirement is that theserver understands the time encoding used, and is able to schedule thespecified event for the appropriate moment in time. In the case wherepreviously stored data are read back from a file, a mechanism such asthe SMPTE time code, which specifies time in streaming video files, canbe used to specify time.

After a time base has been initialized, it can be manipulated throughcommands to the server. Time can be moved around, for instance spedforward or backward, or jumped to a specific new time in the time base.In the XML example, for instance, use of the CONTINUOUS=“false”attribute-value in the SYNCH tag means that time on the time base shouldjump to the new value at the specified UTC time. Similarly,CONTINUOUS=“true” means that that time should be quickly moved forwardor back to the new time base time at the specified UTC time. Note that adiscontinuous jump in time means that events between the current timeand the new time are not delivered to the enhancement, whereas acontinuous jump means that all events between the current time and thenew time should be delivered immediately once the jump occurs.

The synthetic time base may also be paused at any time through a commandto the server. Such a command, in XML format, could appear as

<CAROUSEL NAME=“kqed” VIRTUAL=“trigger512”> <PAUSEUTC=“2001-12-20T02:30:00”> </CAROUSEL>

Pause times may be specified in absolute time (UTC, GMT, or other), inthe synthetic time base of the substream, or through any other mechanismthat can be used to identify a time uniquely to the server.

The time base associated with a substream may also be destroyed withoutdestroying the substream itself. This may be useful when the substreamis being used to carry items that are not time dependent, or when thesystem wishes to put the time base into a temporary suspended state. Inthe XML example, such a command to the server could be:

<CAROUSEL NAME=“kqed” VIRTUAL=“trigger512”> <UNSYCHUTC=“2001-12-20T04:30:00”> </CAROUSEL>

As with pause times, unsync times may be specified in absolute time, thesynthetic time base of the server, or any other mechanism that uniquelyidentifies time to the server.

It is to be noted that these pause, resume, create, destroy, synch, andunsynch concepts apply to the synthetic time base attached to a datasubstream. The state of the enhancement itself need not be changed.

The protocol to transmit this timing information to the client consistsof time pulses that are associated with the data substream. These timepulses are delivered regularly to the client, and consist of the currenttime on the substream's time base.

Information about coming events, such as upcoming time transitions, mayalso be transmitted to the client. Such information should betransmitted multiple times to the client because losses within thetransmission mechanism can prevent receipt of any particular piece ofdata.

For instance, if these time pulses where transmitted as strings, atypical time pulse looks like:

-   -   trigger512/T″25000

where trigger512 indicates the data substream and where T″ the standardtime pulse of a running time stream with a time of 25000 ms.

Similarly, time pulses associated with a paused stream might have aspecial code, such as:

-   -   trigger512/Z″25000

which indicates that the time base associated with data substreamtrigger512 has been paused at time 25000 ms. Communicating thedistinction between paused and unpaused time bases to the client isespecially important when the client may start receiving data on thesubstream at any time: the internal state of the client may be set verydifferently if it first receives a paused time pulse rather than astandard time pulse.

Time pulses associated with upcoming time base changes have twoassociated times, i.e. a time at which the change takes place, and a newtime to set the time base at that time. For instance,

-   -   trigger512/D″25000-35000

means that the time base should jump forward 10 seconds (35000 ms−25000ms) at time 25000 ms on trigger512's time base.

It is important that the protocol for communicating between the clientand the server does not necessarily include an absolute time. Thisallows the timing information to be entirely self-contained, whichallows it to recorded along with the enhancement, and played back at alater time without any loss of coherence. It is expected that thismechanism is of use not just in synchronizing enhancements with otherdata sources, but also in “digital video recorder” (DVR) and video ondemand (VOD) applications, where the time line associated with the mediacan be manipulated by the end user. In cases where the data streams arereplayed at a later date, the data source could directly operate onfiles, or could be injecting data into a real-time recording system.

FIG. 5 shows the multiplexing of data and timing information for twodata substreams, subStream A and subStream B, on a single data stream150. Note that the main data stream may contain data 151 that are notassociated with a substream, as well as data 152, 153 that are specificto each contained substream.

Individual items that are to be delivered to the enhancement atspecified times according to the synthetic time base can be sentmultiple times from the server, and are sent before they are scheduledto be delivered. This mechanism ensures that the data are available tothe client before they are needed by the enhancement, and in the case ofa glossy transmission medium, guarantees as much as possible that thedata are actually received by the client.

On the client side, the client uses these pulses to recreate theinternal time base associated with the data substream. Data are notguaranteed to be transmitted to the client without loss, andtransmission delays within the client itself or the in broadcast mediumare likely to cause jitter in the reception of these time pulses. Theclient applies an appropriate smoothing algorithm, for instance acalculated average over eight time pulses, to recreate the original timebase from the incomplete data that it receives.

The actual data that are to be transmitted according to the internaltime base are sent along with an identifier of the data substream towhich they belong, and the time in that substream's time base in whichthey are to be presented to the enhancement. Again, to ensure correctreception, these data are transmitted early to the client, and may betransmitted multiple times.

For instance, the server can be told to place a simple name value pairin the data substream with the following XML command:

<CAROUSEL NAME=“kqed” VIRTUAL=“trigger512”> <ITEM> < SCHEDULESTART=“25000” DELIVER=“35000” INTERVAL=“1000”> <DATA NAME=“question”VALUE=“What is the price of tea in China?”> <DATA NAME=“answer1”VALUE=“1500 yuan/kg”> <DATA NAME=“answer2” VALUE=“2500 yuan/kg”> <DATANAME=“answer3” VALUE=“More than I can afford”> </ITEM> </CAROUSEL>

In this example, data consisting of four name value pairs are to be putinto the data substream trigger512 in main data stream kqed. The dataare placed in the stream at the data substream's internal time of 25000ms, for delivery to the enhancement 10 seconds later, at internal timeof 35000 ms. These data are to be sent every second (1000 ms).

Although this example uses data only in the form of string name valuepairs, this is not intended as a restriction. Data delivered in thisfashion can be arbitrary: they need only be associated with a particularsynthetic time and a data substream carrying the synthetic time base.

On the client side, the enhancement or enhancement service may choose tolisten to any of the data substreams that it wishes, and need notreceive information for the others. In an MPEG system, this can beaccomplished by adding additional descriptors to the substream that areused to differentiate it from other substreams, or by adding identifiersto the individual items on the substream that reflect the substream towhich those items are associated. Other mechanisms are possible withother broadcast media.

In reference to the enhancement system as presented above, thismechanism can be used to guarantee the delivery of any items on thethree main data streams at a particular time in a substream's time base.For instance, a data substream can be set up on the Trigger stream foran enhancement using that stream. The enhancement then receives timesensitive information at a particular time within its own time base.Furthermore, multiple enhancements can share the same Trigger stream,but each enhancement is associated with a different substream withinthat stream. In this case, each enhancement receives trigger data at atime specified in the time base of its own substream, which may or maynot correspond to the time bases of substreams of the otherenhancements. Time in two enhancements, as defined by the time base oftheir associated trigger streams, need not even flow at the same pace inthe two enhancements, even if they are running simultaneously.

Because of the programmatic and dynamic nature of the substreams,enhancements from multiple independent and mutually unaware sources cancoexist in a single system. For instance, an enhancement on the ESPN canbe set up and scheduled without any need for knowledge of an enhancementassociated with an ad break. They both reference their own internal timebases, create and destroy their own internal substreams, and receiveonly their own data, even though they are running at the same time.

Announcements and Preloading

The announcement stream is used to transmit information concerning theavailability of enhancements on the system, as well as information abouthow to initialize the enhancement and what resources it needs to run.Information transmitted by the announcements may consist of times,relative to the receipt of the announcement, that the enhancement shouldbegin and end; times, also relative to the receipt of the announcement,during which the enhancement should be paused and its interface removedfrom the users perception (this feature is useful for insertingadvertisements into the enhancements, for example); a URL or otherreference to the main program or resource bundle associated with theenhancement; a URL or other reference to an item to be used to promptthe user to start the enhancement; and a URL or other reference tocontent intended to be loaded before the enhancement begins.

A list of announcement items is contained in Table 1 below. It is notmeant to be an exhaustive list of all possible items in an announcement,nor are all of the items considered to be mandatory.

TABLE 1 Announcement Items Item Name Purpose startTime time in secondsafter receipt of the announcement at which prompting for the enhancementshould begin. If there is no prompt, time at which the enhancementshould begin pauseTime time in seconds after the receipt of theannouncement at which the enhancement should be paused, and all UI madeinvisible to the user resumeTime time in seconds after the receipt ofthe announcement at which the enhancement should be resumed from thepause state. UI should be made visible again at this time killTime timein seconds after the receipt of the announcement at which theenhancement should be terminated Id identifier for the enhancement orgId identifier for the organization profile profile for the run timeenvironment of the enhancement - HTML3.2, HTML4.0, Java, etc urlreference to the initial enhancement content to be loaded and run promptreference to image used to prompt user to enter enhancement, may alsospecify system specific default prompt promptSize size of the prompt(width, height) promptLoc location of the prompt (x, y) promptDurationlength of time to display the prompt promptAcceptKey Key on the clientused to accept the prompt memory amount of client side memory requiredby the announcement triggerUrl reference to stream used for triggersassociated with the enhancement. Used by the enhancement, in conjunctionwith a client side API for listening to streams, to listen to thetrigger stream preloadUrl comma delimited list of references to contentwhich must be loaded on the client before the enhancement is run andbefore any prompting or other indication of the availability of theenhancement is made to the user.

On the client, the software listening to the announcements may launch anannounced enhancement, per the information contained in theannouncement. The listening software does not have the option oflaunching an enhancement that is not contained in the announcement, anddoes not have the option of launching the enhancement in a wayinconsistent with the announcement. It may, however, decide not tolaunch the enhancement at all, due to insufficient resources on theclient, an inability to find the enhancement data, policies set by othersoftware running on the client, user preferences, or any other reason.

Interactions with Other Client Sub-Systems

The invention also comprises a technique for allowing enhancements tointeract with other modules in the client in a secure manner.Enhancements are typically programs or program fragments, either ininterpreted form or in abstract byte code form, and can be considered asa program in a general purpose computing environment. In many systems,there is a privilege level that controls what actions a program mayperform. For example, an enhancement may be allowed to open a newwindow, or may not. The program makes function calls to various systemresources, which can be implemented in native software, interpretedcode, or abstract byte code. The privileges of the program are one ofthe factors the system software uses to determine whether the program isallowed access to those functions. The privilege level at which aprogram run is associated with the program's execution context. Aproblem arises when a lower privilege program needs to use functions forwhich it is not privileged. Typically, it is desired to give the lowerprivileged program access to a very restricted subset of the privilegedfunctionality, which is why simply granting a higher level of privilegeto the lower privileged program is undesirable.

As an example of this problem in common usage today, many networkedapplications, such as Java and JavaScript, make use of an Access ControlList (ACL), which specifies URLs, or URL domains, of trusted content.This list is typically transmitted to the client by network operators,and decisions about whether to allow content to access privilegedfunctionality are made by comparing the URL of the content with theprivileged URLs in the ACL list.

The difficulties of this approach are two fold:

1. It is not flexible enough. Typical ACL systems provide only a fewlevels of pre-assigned privileges, and they do not correspond well tothe needs of enhancement systems.

2. Maintaining the ACL list is problematic for the network operators.Because the ACL list is a set of URLs with corresponding privileges,each time new privileged content is made available to the system, theACL list may need to change. This is especially troublesome indistributed system, such as enhanced TV, where the people responsiblefor distributing the content, i.e. the broadcasters, are not the peopleresponsible for maintaining the ACL list, i.e. the cable, terrestrial,or satellite TV operators.

FIG. 6 is a block diagram that illustrates this problem of accessing aprivileged function from an unprivileged program. In this situation, anunprivileged program 20 has a function foo 23 that calls 24 a functionbar 24 which is a privileged function 26 that resides in a privilegedprogram 22. Here, the context of the unprivileged program cannot accessthe privileged function.

The invention solves this problem by allowing a privileged program topublish functions which the implementer wishes to make accessible toless privileged programs. The mechanism for the publication is stringbased, that is, the privileged program specifies by name the functionthe implementer wishes to publish, the name under which the publishedfunction is to be known, and one or more strings corresponding to theprivilege levels that a subscribing program must have to access thefunction. When an unprivileged program calls a privileged functionpublished in this way, the context of the unprivileged program istemporarily swapped with the context of the privileged program.

FIG. 7 is a block diagram that illustrates the technique for accessing aprivileged function from an unprivileged document according to theinvention. As shown on FIG. 7, the program swaps contexts 27, 29, sothat the thread of control 28 temporarily runs in the privilegedcontext.

Permissions to access these published functions are granted by highlyprivileged programs. The highly privileged program associates aprivilege with a function. Typically, this is a comma-delimited stringof custom privileges with the context of a program with lowerprivileges, although the invention also allows standard privileges to beso granted. If any of the privileges granted to the lower privilegeprogram matches any of the privileges that a published function waspublished with, the lower privilege program may access the publishedfunction.

By allowing both standard privileges and custom privileges to begranted, the invention not only creates arbitrary flexibility inprivilege levels, but can also subsume the standard privilegemechanisms. In practice, for instance, this mechanism can be used toeliminate the need for the ACL list currently used by network operators.

In use, this feature can take the form:

navigator.exports.exportFunction(“grpName”, “aliasName”, function,“privString”),

-   -   where:    -   “grpName” is the name of an arbitrary grouping of similar        services under which the function is exported,    -   “aliasName” is the name by which the exported function is known,        and    -   “privString” is a comma separated list of privileges needed to        access the function (any privilege in the list grants access).

For instance, “navigator.exports.exportFunction(“etv”, “myPriv”,privFuncl, “etv”) exports the function privFuncl as:

-   -   navigator.exports.etv.myPriv( )

and only contexts with the custom “etv” privilege are allowed to use it.

In the preferred embodiment of the invention, the feature of exportingfunctions is a highly privileged operation. Thus, one can also unexporta previously exported function.

For the function:

-   -   navigator.exports.unexportFunction(srvName, funcName)

When a function has been unexported, or if a context lacks permissionsto see the function, the function object appears null.

A symmetric problem to the above occurs when a privileged program wishesto use functions defined by unprivileged programs. Typically this occurswhen the standard API defined for allowing the two programs to interactrequires a callback to be provided by the less privileged program to themore privileged program.

In this case, the more privileged program is expected to call thecallback of the less privileged function. Because the callback is partof the less privileged program, it cannot be trusted in the same way asthe more privileged program. However, if the callback is called directlyfrom the more privileged program, it is run in the context of the moreprivileged program, and has that program's trust level and securityaccess.

The solution to this problem is essentially similar to the solution forthe symmetric case. A special call is made which allows the callback tobe run in the less privileged context. FIG. 8 is a block diagram thatillustrates a technique for signaling between privileged andunprivileged contexts without granting security access according to theinvention. In this case, however, there is no need for custom definedprivileges because the higher priority program is assumed to beprivileged enough to invoke lower privilege functionality.

A sample implementation in JavaScript is as follows:

-   -   navigator.exports.invokeCallback(window, this, func)

Where:

-   -   window defines which context to use for the callback    -   this is the “this” when the function is called    -   func is the function to be invoked.

The function invokeCallback( ) returns an object with two properties:

-   -   result—the return value of the function called    -   failed—whether or not that function threw a JS Error.

In JavaScript, capturing the return values in this way allows thecaller's thread not to be aborted by an error made by the callee.

This security and privilege-granting mechanism is applicable to otherprogramming languages as well, including Java andother-bytecode-oriented systems.

Typically, Enhanced TV is thought of in terms of current televisiondistribution systems, such as analog broadcast or digital cable andsatellite. It is equally applicable to other distribution mediums anddisplay devices over which video and audio can be delivered, includingIP-based systems (such as video over DSL) and video over wirelessnetworks (such as distribution over a 3G network to a mobile phone.)

Although the invention is described herein with reference to thepreferred embodiment, one skilled in the art will readily appreciatethat other applications may be substituted for those set forth hereinwithout departing from the spirit and scope of the present invention.Accordingly, the invention should only be limited by the Claims includedbelow.

The invention claimed is:
 1. A method comprising: receiving at acomputer server a plurality of data substreams, each data substreamidentifying the availability of an enhancement of a plurality ofenhancements presentable with one or more videos by a client device, andeach data substream associating a set of content of a plurality of setsof content with the enhancement identified by the data substream; andmultiplexing the plurality of data substreams into a data stream,wherein one data substream of the plurality of data substreamsidentifies certain content, in the set of content associated by the onedata substream, which must be preloaded by the client device prior torunning the enhancement identified by the one data substream.
 2. Themethod of claim 1, further comprising transmitting the data stream tothe client device.
 3. The method of claim 2, further comprising:receiving each set of content at the computer server; multiplexing thesets of content into one or more content streams; and transmitting theone or more content streams to the client device.
 4. The method of claim1, wherein one set of content of the plurality of sets of contentincludes machine-readable instructions executable by the client device,that when executed, cause the client device to present the enhancementassociated with the one set of content synchronously with the one ormore videos.
 5. A method comprising: receiving at a computer server aplurality of data substreams, each data substream identifying theavailability of an enhancement of a plurality of enhancementspresentable with one or more videos by a client device, and each datasubstream associating a set of content of a plurality of sets of contentwith the enhancement identified by the data substream; and multiplexingthe plurality of data substreams into a data stream, wherein each set ofcontent is associated with a mutually independent enhancement receivedfrom a different provider.
 6. A method comprising: receiving at acomputer server a plurality of data substreams, each data substreamidentifying the availability of an enhancement of a plurality ofenhancements presentable with one or more videos by a client device andeach data substream associatin a set of content of a plurality of setsof content with the enhancement identified by the data substreaml;multiplexing the plurality of data substreams into a data stream;receiving the plurality of sets of content at the computer server;multiplexing the plurality of sets of content into one or more contentstrearns; and storing the data stream, the one or more content streams,and the one or more videos on a video-on-demand server.
 7. The method ofclaim 6, further comprising: transmitting the data stream and the one ormore content streams to the client device in response to a request fromthe client device to the video-on-demand server.
 8. A method comprising:receiving at a computer server a plurality of data substreams, each datasubstream identifying the availability of an enhancement of a pluralityof enhancements presentable with one or more videos by a client device,and each data substream associating a set of content of a plurality ofsets of content with the enhancement identified by the data substream;and multiplexing the plurality of data substreams into a data stream,wherein each data substream includes times, relative to the receipt ofthat data substream by the client device, at which the enhancementidentified by that data substream should be presented with the one ormore videos.
 9. A method comprising: receiving at a computer server aplurality of data substreams, each data substream identifying theavailability of an enhancement of a plurality of enhancementspresentable with one or more videos by a client device, and each datasubstream associating a set of content of a plurality of sets of contentwith the enhancement identified by the data substream; multiplexing theplurality of data substreams into a data stream; receiving at thecomputer server a plurality of trigger data substreams, each triggerdata substream indicating a timing state of one of the enhancements, thetiming state of each enhancement being relative to a virtual time baseunique to that enhancement; creating timing information enabling theclient device to advance each virtual time base independent of everyother virtual time base; multiplexing the plurality of trigger datasubstreams and the timing information into a trigger stream; andtransmitting the trigger stream to the client device.
 10. A methodcomprising: receiving at a computer server a plurality of datasubstreams, each data substream identifying the availability of anenhancement of a plurality of enhancements presentable with one or morevideos by a client device, and each data substream associating a set ofcontent of a plurality of sets of content with the enhancementidentified by the data substream; multiplexing the plurality of datasubstreams into a data stream; receiving the data stream at the clientdevice; monitoring the data stream at the client device for theavailability of each enhancement; and launching one or more of theenhancements at the client device based on the monitored availabilityidentified in the data substreams.
 11. A system comprising one or morecomputers storing one or more programs that when executed by the one ormore computers cause the system to: receive a plurality of datasubstreams, each data substream identifying the availability of anenhancement of a plurality of enhancements presentable with one or morevideos by a client device, and each data substream associating a set ofcontent of a plurality of sets of content with the enhancementidentified by the data substream, multiplex the plurality of datasubstreams into a data stream, wherein one data substream of theplurality of data substreams identifies certain content, in the set ofcontent associated by the one data substream which must be preloaded bythe client device prior to running the enhancement identified by the onedata substream.
 12. A system comprising one or more computers storingone or more programs that when executed by the one or more computerscause the system to: receive a plurality of data substreams, each datasubstream identifying the availability of an enhancement of a pluralityof enhancements presentable with one or more videos by a client device,and each data substream associating a set of content of a plurality ofsets of content with the enhancement identified by the data substream;multiplexing the plurality of data substreams into a data stream;receive the plurality of sets of content at the computer server;multiplex the sets of content into one or more content streams; andstore the data stream, the one or more content streams, and the one ormore videos on a video-on-demand server.
 13. The system of claim 12,wherein the one or more programs, when executed by the one or morecomputers, further cause the system to: receive a plurality of triggerdata substreams, each trigger data substream indicating a timing stateof one of the enhancements, the timing state of each enhancement beingrelative to a virtual time base unique to that enhancement; createtiming information enabling the client device to advance each virtualtime base independent of every other virtual time base; multiplex theplurality of trigger data substreams and the timing information into atrigger stream; and store the trigger stream on the video on demandserver.
 14. The system of claim 13, wherein each data substreamassociates one of the trigger substreams with the enhancement identifiedby that data substream.
 15. The system of claim 12, wherein the one ormore programs, when executed by the one or more computers, further causethe system to: transmit the data stream, the one or more contentstreams, and the one or more videos to the client device in response toa request from the client device to the video-on-demand server.
 16. Asystem comprising one or more computers storing one or more programsthat when executed by the one or more computers cause the system to:receive a plurality of data substreams each data substream identifyingthe availability of an enhancement of a plurality of enhancementspresentable with one or more videos by a client device, and each datasubstream associating a set of content of a plurality of sets of contentwith the enhancement identified by the data substream; and multiplex theplurality of data substreams into a data stream, wherein each datasubstream includes times, relative to the receipt of that data substreamby the client device, at which the enhancement identified by that datasubstream should be presented with the one or more videos.
 17. A systemcomprising: a server including one or more computers storm one or moreprograms that when executed by the one or more computers cause theserver to: receive a plurality of data substreams, each data substreamidentifying the availability of an enhancement of a plurality ofenhancements presentable with one or more videos, and each datasubstream associating a set of content of a plurality of sets of contentwith the enhancement identified by the data substream, and multiplex theplurality of data substreams into a data stream; and a client device,wherein the client device includes a client computer storing one or moreclient programs that when executed by the client computer, cause theclient device to: receive the data stream from the server, monitor thedata stream for the information identifying the availability of eachenhancement, and launch one or more of the enhancements based on themonitored availability identified in the data substreams.
 18. One ormore non-transitory computer readable memory having one or more programsstored thereon, that when executed by one or more computers, cause asystem to: receive a plurality of data substreams, each data substreamidentifying the availability of an enhancement of a plurality ofenhancements presentable with one or more videos by a client device, andeach data substrearn associating a set of content of a plurality of setsof content with the enhancement identified by the data substream,wherein at least one of the data substreams identifies client resourcesrequired to run the enhancement identified by the at least one datasubstream; and multiplex the plurality of data substreams into a datastream.
 19. One or more non-transitory computer readable memory havingone or more programs stored thereon, that when executed by one or morecomputers, further cause a system to: receive a plurality of datasubstreams, each data substream identifying the availability of anenhancement of a plurality of enhancements presentable with one or morevideos by a client device, and each data substream associating a set ofcontent of a plurality of sets of content with the enhancementidentified by the data substream; multiplex the plurality of datasubstreams into a data stream; receive a plurality of trigger datasubstreams, each trigger data substream indicating a timing state of oneof the enhancements, the timing state of each enhancement being relativeto a virtual time base unique to that enhancement; create timinginformation enabling the client device to advance each virtual time baseindependent of every other virtual time base; multiplex the plurality oftrigger data substreams and the timing information into a triggerstream; and transmit the data stream and the trigger stream to theclient device.
 20. The one or more non-transitory computer readablememory of claim 19, wherein each data substream associates one of thetrigger substreams with the enhancement identified by that datasubstream.